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Michael  Pryce — is  a  Lecturer  in  defence  acquisition  at  Cranfield  University.  He  teaches  across  a 
range  of  subjects,  including  the  use  of  costing  in  acquisition  and  planning  at  the  master’s  level.  His 
research  focuses  on  the  use  of  cost  as  a  design  tool.  He  is  currently  leading  a  second  costing 
research  project  with  colleagues  from  UCL,  Imperial  College,  and  DSTL  and  developing  the  Low  Cost 
by  Design  (LCxD)  network.  Prior  to  joining  Cranfield,  Dr.  Pryce  worked  at  Manchester  Business 
School,  where  he  carried  out  two  NPS-funded  projects  looking  at  the  role  of  costing  and  business 
models  in  operations  and  support,  [m.pryce@cranfield.ac.uk] 

Abstract 

A  program  of  current  research  funded  by  the  United  Kingdom  (UK)  research  councils  and 
supported  by  the  UK  Defence  Science  and  Technology  Laboratory  (DSTL)  is  reported.  The 
work  involves  pioneering  data  collection,  analysis,  and  tool  development  to  support  future  air 
combat  systems.  The  role  of  a  community  of  users  and  developers  of  the  data  and  tools  is 
reported,  as  well  as  the  underpinning  philosophy  of  the  work  and  future  prospects  for  its 
wider  application. 

Summary 

The  United  Kingdom  National  Audit  Office  Major  Projects  Report  2014  states  that 
“Project  teams  continue  to  be  over-optimistic  in  their  forecasts  of  both  procurement  and 
support  costs”  and  that  “Budgets  set  using  over-optimistic  forecast  costs  could  result  in 
overall  budgets  for  procurement  and  support  being  significantly  understated”  (Great  Britain, 
National  Audit  Office,  2015).  Correcting  the  causes  of  such  failings  is  the  aim  of  this 
research,  with  the  outputs  being  intended  for  use  by  a  wide  range  of  stakeholders. 

The  research  will  lead  to  enhanced  methods,  tools,  and  understanding  of  costing  in 
defence.  This  will  be  realised  through  the  creation  of  a  database  of  historic  defence  costs,  a 
set  of  project  histories,  and  other  contextual  information.  An  associated  viewer  tool,  and  a 
rating  and  health  check  tool  for  use  by  practitioners,  will  be  developed. 

The  project  will  take  work  that  has  previously  been  at  a  largely  conceptual  level 
through  to  initial  field  trials  and  validation.  Later  phases  leading  to  full  deployment  are  also 
planned. 

The  end  users  for  the  work  will  be  in  the  Aerospace,  Defence,  and  Manufacturing 
sectors.  The  successful  delivery  of  this  project  will  allow  users  to  easily  organise  and  access 
key  contextual  information  that  will  support  more  accurate  forecasting  of  project  costs  and 
schedules.  This  will  allow  more  efficient  decision  making. 

During  the  period  of  active  research,  the  DSTL  provides  the  initial  customer  and 
significant  support  in  informing  the  development  of  the  work.  The  project  is  currently  planned 
to  run  July  1, 2015,  to  June  30,  2016. 

Project  Overview 

The  research  involves  the  collation  of  a  large,  varied  data  set  on  historic  defence 
project  costs,  as  well  as  associated  qualitative  information  in  the  form  of  project  histories. 
The  work  will  lead  to  a  wider  understanding  of  the  root  causes  of  cost  in  defence,  as  well  as 
enhancing  knowledge  of  how  to  improve  the  accuracy  of  early  stage  project  cost  estimates. 

The  research  findings  will  be  linked  semantically  and  interpreted  through  a  viewer 
tool  that  relates  data  to  projects  interactively,  based  on  HTML5,  allowing  users  to 
contextualise  cost  data.  A  second  tool  to  allow  project  teams  to  carry  out  costing  “health 
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checks”  is  also  being  developed.  This  will  allow  users  of  the  viewer  to  focus  their  interests 
on  the  data  and  examples  that  matter.  The  project’s  top  level  structure  is  shown  in  Figure  1 . 
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Figure  1.  Top  Level  Structure 

Note.  Tools  are  shown  in  blue. 

The  project  will  focus  its  initial  data  collection  on  a  number  of  combat  air  systems. 
These  have  already  been  partly  researched  by  the  team  in  prior  work,  and  therefore  the 
context  is  reasonably  well  understood.  This  will  allow  the  work  to  focus  on  more  in-depth 
data  gathering  and  the  development  of  the  key  outputs.  A  key  focus  of  the  work  will  be  in 
tackling  variety  in  the  data  collected,  one  of  the  “3  Vs”  of  Big  Data.  The  outline  programme  is 
shown  in  Figure  2. 
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Figure  2.  Programme  of  Work 

Note.  Project  phases:  1  =  Data  collection,  2  =  Data  presentation  and  evaluation,  3  =  Initial 
development  and  testing  of  viewer  tool,  4  =  Development  of  health  check  tool,  5  =  User  trials, 

6  =  Revision  and  expansion  plan. 

The  project  reviews  are  carried  out  by  Professor  David  Kirkpatrick,  who  acts  as  an 
“external  examiner”  of  the  thinking  behind  the  project,  as  well  as  contributing  his  own 
extensive  experience  in  this  field  to  help  guide  the  work  and  its  application. 

End-Users 

User  trials  involve  a  range  of  potential  end  users,  with  informal  meetings  held  ahead 
of  the  planned  trials.  The  initial  end  users  for  the  work  are  in  the  defence  sector,  including 
the  UK  Ministry  of  Defence  (MOD),  particularly  Defence  Equipment  &  Support  and  Air 
Command,  and  UK  defence  companies  such  as  BAE  Systems  and  Rolls  Royce.  The  work 
also  has  the  potential  to  meet  already  articulated  needs  from  the  Australian  Department  of 
Defence  and  the  Norwegian  MOD. 


ACQUISITION  RESEARCH  PROGRAM: 
CREATING  SYNERGY  FOR  INFORMED  CHANGE 


-284- 


The  successful  delivery  of  this  project  will  allow  end  users  to  easily  organise  and 
access  key  contextual  information  that  will  support  more  accurate  forecasting  of  defence 
project  costs  and  schedules,  especially  complex  and  novel  ones  such  as  unmanned  combat 
systems. 

Technology  Readiness 

The  project  will  take  work  that  has  previously  been  at  a  largely  conceptual  level  and 
develop  it  to  initial  field  trials.  For  the  tools  discussed,  this  would  equate  to  going  from  TRL  1 
to  TRL  6. 

Outputs 

The  project  is  intended  to  lead  to  facilitated  workshops,  the  creation  of  a  network  of 
cost  researchers  and  end  users,  a  number  of  high-quality  academic  publications,  and  the 
development  of  a  database  of  defence  costs  and  project  histories. 

The  findings  will  be  used  to  develop  teaching  material  for  the  Cost  Estimating  and 
Planning  module  of  Cranfield  University’s  Defence  Acquisition  Management  MSc.  Students 
on  the  MSc.  are  mainly  senior  managers  from  potential  end  user  organisations  inside 
defence.  The  work  will  also  be  relevant  to  the  UK  MOD’S  Financial  and  Military  Capability 
management  environment,  which  provides  the  high-level  planning  framework  for  UK 
defence  acquisition. 

Outcomes 

Beyond  the  completion  of  the  proof  of  concept  work  represented  by  this  project,  a 
number  of  additional  outcomes  are  being  pursued: 

•  Licensing  of  tools  in  a  wider  range  of  user  organisations 

•  Database  to  be  openly  accessible  within  the  costing  community 

•  The  development  of  new  decision  making  approaches  in  the  pre-concept 
phase 

•  Better  understanding  of  organisational  learning  and  corporate  memory 

•  Links  to  existing  costing  methods  that  rely  on  data  similarity 

•  Further  development  to  other  domains  beyond  air,  and  sectors  outside 
defence 

Rationale  for  the  Research 

For  many  years,  the  UK  has  struggled  to  maintain  an  extensive,  coherent  and 
useable  set  of  project  costing  data  for  the  development  of  major  new  platforms. 
Organizational  changes  within  acquisition,  contracting  strategies  and  the  high  turnover  in 
defence  project  staff  have  contributed  to  both  the  loss  of  data  and  low  numbers  of 
experienced  staff  able  to  interpret  such  data  (Gray,  2009;  Levene,  201 1 ). 

These  problems  have  been  exacerbated  by  the  focus  in  the  last  15  years  on  the 
support  and  sustainment  of  existing  equipment.  This  has  led  to  the  widespread  use  of 
commercial  costing  tools  that,  while  acceptable  for  the  costing  of  in-service  platforms,  are  of 
less  use  for  the  development  of  highly  novel  platforms,  especially  when  the  proprietary 
nature  of  the  tools’  databases  means  that  it  is  difficult  to  make  allowances  and  adjustments 
for  novelty.  The  “bottom  up”  approach  to  costing  works  well  for  established  designs,  but  is 
almost  impossible  for  novel  ones. 

At  the  same  time,  the  understanding  of  the  need  to  cost  “through  life”  that  has  been 
engendered  by  the  focus  on  support  and  sustainment  has  made  the  challenges  of 
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generating  realistic  costs  at  an  early  stage  of  a  project,  when  major  commitments  may  be 
entered  into,  increasingly  important,  though  no  less  difficult. 

While  prior  work  has  shown  that  the  operation  and  support  costs  are  not  necessarily 
fixed  early  in  the  life  cycle  (Pryce,  2011),  the  need  to  capture  their  likely  range,  as  well  as  to 
ensure  that  the  full  range  of  issues  outlined  by  Pugh  (1992)  for  development  and  production 
are  also  addressed,  indicated  that  current  data  sets  and  methods  in  the  UK  were 
inadequate.  The  research  was  developed  on  the  basis  of  creating  a  new  method  with 
related  data  and  tools  that  can  be  generated  dynamically  and  sustained  by  a  wide 
community  over  a  long  period  of  time. 

This  new  approach  is  not  intended  to  deliver  a  model.  Rather,  it  is  aimed  at  creating 
a  heuristic  or  “therapeutic”  set  of  data,  tools  and  the  necessary  understanding  to  use  them 
that  will  enable  better  decisions  to  be  made  by  project  teams  in  defence  acquisition. 
Philosophically,  it  differs  from  many  approaches  that  are  currently  used,  but  aims  to  do  so  in 
a  complementary  way.  Such  aspects  will  be  explored  through  later  work  on  a  related  project, 
which  will  seek  to  add  a  degree  of  modelling  capability  to  some  of  the  data  being  generated, 
and  the  idea  of  Bayesian  belief  networks  will  also  be  explored  in  this  additional  work. 

The  ultimate  objective  is  to  help  support  the  “smart  customer”  inside  the  UK  MOD, 
and  to  provide  an  approach  that  promotes  dialogue  between  government  and  industry. 

The  Project  Described 

The  project  is  made  up  of  a  number  of  activities  and  phases  as  shown  in  Figures  1 
and  2  above.  A  more  detailed  description  of  these  is  given  below  in  order  to  understand  the 
intent  of  each. 

Figure  3  shows  how  the  work  begins  with  field  research.  This  consists  of  the 
extensive  collation  of  information  and  data  from  archives  across  the  United  Kingdom.  The 
National  Archive  at  Kew,  BAE  Systems  Fleritage  archives,  the  Royal  Air  Force  Museum, 
Brooklands  Museum,  and  others  are  the  sources  of  a  large  amount  of  historic  material  that 
has  been  lost  to  the  United  Kingdom  MOD  over  the  years.  A  key  part  of  the  research  is  not 
just  the  collection  of  this  data,  but  the  central  recording  of  the  location  and  source  of  the 
material  from  the  archives.  Frequently,  information  from  one  source  helps  make  sense  of 
information  from  another.  It  is  these  cross  connections  that  provide  one  of  the  key  strengths 
of  the  research. 
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Figure  3.  Research 

The  material  obtained  from  the  research  is  broken  down  into  two  basic  elements 
(Figure  4).  Qualitative  information  is  used  to  develop  project  histories  for  past  platforms. 
These  are  not  simple  case  studies,  but  contextually  rich  histories  of  past  projects.  They  can 
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be  used  as  stand-alone  histories  for  one  project  or  platform  or  related  to  others  to  give  a 
programme  or  portfolio  view. 
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Figure  4.  Project  Histories  and  Data  Base 

The  project  histories  can  also  be  related  to  the  quantitative  data  sourced  from  the 
research.  This  is  data  in  both  raw  and  semi-processed  form.  One  of  the  key  methods  in  the 
generation  of  the  recorded  data  is  its  connection  to  its  historic  context  to  ensure  factors  such 
as  exchange  rates,  inflation,  etc.  are  properly  accounted  for,  and  that  suitable  recording  of 
the  derivation  of  the  presented  data  ensures  it  is  fully  explicable  to  end  users. 

The  end  users  explore  the  histories  and  data  obtained  from  the  research  using  a 
viewer  tool.  This  is  intended  to  provide  their  main  method  of  generating  a  knowledge  base 
for  their  own  immediate  needs  (Figure  5).  The  viewer  tool  acts  as  the  main,  IT  based 
resource  that  project  teams  can  utilise  over  time  to  help  them  work  with  the  data  and 
histories.  It  is  intended  to  show  the  data,  histories  and  associated  project  timeline 
information  in  a  single  screen.  A  conceptual  version  is  shown  in  Figure  6,  and  a  mock-up  is 
shown  in  Figure  7. 
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Figure  5,  Viewer  Tool  and  Knowledge  Base 
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Figure  6.  Viewer  Tool  Concept 


2^  August  1985  *  Turin  Agreement 

Armament  Directors  from  UK,  FRG  and  Italy 
agree  to  proceed  with  Eurofighter. 

France  withdraws  from  programme 

Spa  in  later  signs  hut  is  still  considering 
Rafale  at  this  point 

Feasibil  ity  studies  had  not  shown  a  single, 
joint  design  that  was  acceptable  to  all  five 
nations,  but  the  three  signatories  did  have 
a  common  design  (9.75t)  despite  UK 
preference  for  an  lit  aircraft  (Refs:  Flight 
article,  NAFREMfile) 
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Figure  7.  Viewer  Tool  Mock-Up  (Indicative  Data) 

The  viewer  tool  is  being  developed  using  HTML5.  This  is  intended  to  make  it 
platform  agnostic  and  “future  proof.”  A  search  function,  using  meta-tagging  and  standard 
web  search  engine  methods,  allows  intuitive,  adaptable  use  with  minimum  functional 
training. 
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HTML5  will  also  allow  for  development  of  the  viewer  tool  by  project  teams  using  in- 
house  or  external  resources  that  only  require  common,  standard  technologies.  Such  an 
approach  also  means  that  the  viewer  tool  can  run  in  secure/shared/stand-alone  modes 
defined  by  the  user’s  own  IT  environment.  Desktop/tablet  use  is  possible,  as  is  networked 
and  online  use. 

Before  using  the  viewer  tool,  the  data  and  histories  that  are  most  relevant  to  the  user 
are  identified  through  a  health  check  tool.  This  is  intended  to  identify  the  characteristics  of 
the  user’s  current  project  and  to  help  map  this  across  to  the  histories  and  data  shown  by  the 
viewer  tool  (Figure  8). 


Figure  8.  Health  Check  Tool  and  Project  Rating 

The  health  check  tool  uses  qualitative/subjective  inputs  and  relies  on  “polling”  of  the 
user  project  team  to  identify  the  characteristics  of  their  current  project.  This  is  intended  to 
capture  diverse  views  using  a  rating  system  conceptually  similar  to  the  “Cooper  Harper” 
style  hierarchy  used  for  piloting  aircraft.  Alphanumeric  outputs  are  used  to  identify  “true 
likeness”  projects  from  which  learning  can  occur  and  to  help  identify  if  the  context  of  the 
user’s  project  is  similar  to  the  one(s)  explored  using  the  viewer  tool.  These  activities  are 
usually  carried  out  through  an  on-site  facilitation  with  the  user  (Figure  9). 
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Figure  9.  Context  Check  and  Lessons  Learned 


Discussion 

The  description  of  the  project  illustrates  how  the  project  is  being  developed  in  an 
essentially  linear  fashion.  However,  this  masks  the  interactive,  incremental  and  innovative 
ways  that  the  tools,  data,  histories  and  basic  research  can  be  used. 

For  users,  their  knowledge  base  can  be  constantly  reinforced  and  updated  by 
repeated  use  of  the  tool,  forming  a  loop  that  links  their  learning  from  the  tool  with  the 
changed  perception  and  performance  of  their  project.  This  can  be  allied  to  adaptation  of  the 
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viewer  tool  to  their  own  specific  needs.  Both  aspects  are  intended  to  go  some  way  to 
compensating  for  the  loss  of  skills  and  data  that  has  happened  in  the  UK  MOD  costing 
community  in  the  last  20  years  with  regard  to  cost  forecasting  for  complex  and  novel 
projects. 

For  the  research  team,  involvement  in  users’  activities  through  data  provision,  tool 
set  up  and  facilitation  helps  to  refine  the  further  development  of  the  research  outputs,  as 
well  as  helping  to  focus  the  collection  of  further  data.  It  has  already  been  found  that  much 
more  data  and  information  exists  in  archives  than  was  expected,  and  that  the  corporate 
memory  on  projects  such  as  the  Eurofighter  Typhoon  and  F-35B  (i.e.,  projects  still  in 
development  but  decades  old)  can  be  usefully  refreshed  using  archival  sources. 

Although  the  work  is  currently  focussed  on  air  combat  systems,  a  second  project, 
funded  by  the  DSTL  (called  Air  Systems  Programme  Data)  is  now  beginning  that  will  start  to 
address  how  an  approach  that  works  in  the  air  domain  can  be  extended,  notably  to  the 
maritime  sector.  This  will  also  widen  academic  participation,  with  the  research  team 
involving  staff  from  University  College,  London  (David  Andrews),  Imperial  College  (Dr. 
Michael  Weatherburn),  and  industry  (Andrew  Dakin,  649  Ltd.). 

The  approach  taken  to  this  project  mirrors  many  of  the  techniques  becoming 
increasingly  prevalent  in  the  emerging  field  of  digital  humanities  (Burdick  et  al.,  2012),  while 
aiming  to  avoid  some  of  the  identified  pitfalls.  As  such,  it  represents  a  new  approach  to 
costing,  in  which  the  subjective  understanding  of  contextualised  information  and  data  is 
used  to  assist  decision  making,  rather  than  the  supposed  objective  evidence  given  by  “hard” 
data. 


As  the  work  continues,  it  is  becoming  increasingly  clear  that  the  information  and  data 
provided  on  a  screen  is  only  the  tip  of  the  iceberg  of  potential  benefits.  Re-building,  through 
digital  humanities  type  techniques,  the  institutional  capacity  to  understand  cost  as  part  of  the 
early  stage  design  process  is  a  trans-disciplinary  activity.  To  benefit  from  it  requires  a 
diverse  set  of  engagements  with  users  in  a  truly  open  and  innovative  manner. 
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Disclaimer 

•  All  views  my  own 

•  Do  not  represent  those  of  other  participants 

•  Do  not  represent  those  of  third  parties 

•  Work  in  progress  -  this  is  last  week's  version! 
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The  problem 

•  Cost  matters 

•  BBP  -  Consciousness  (DARJ) 

•  Leadership  -  FinMilCap  -  flex 

•  Cost  is  a  design  issue  -  Rosetta  stone 

•  Corporate  memory  lacking  -  cost  lens' 
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The  project 

•  Concept/pre  concept  forecasts 

•  RDT&E,  Procurement,  O&S  and  links 

•  Database  of  historic  defence  costs 

•  Project  histories  &  contextual  information 

•  Air  systems  initially  (ASTOVL/UCAV) 

•  Maritime  and  systems  (carrier  strike) 


©  Copyright  Cranfield  University  2016.  All  Rights  Reserved 


Costing  Future  Complex  &  Novel  Projects 


Cranfield 

/  UNIVERSITY 


The  project 

•  Not  a  model  -  complements 

•  Heuristic/'therapeutic'  (blinders  off) 

•  Create  'smart  customer' 

•  Link  to  current  methods 

•  Proof  of  concept 

•  User  trials  (MOD/industry/aerospace  etc.) 
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The  project 
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2nd  August  1985  -  Turin  Agreement 

Armament  Directors  from  UK,  FRG  and  Italy 
agree  to  proceed  with  Eurofighter. 

France  withdraws  from  programme 

Spain  latersigns  but  is  still  considering 
Rafale  at  this  point. 

Feasibility  studies  had  not  shown  a  single, 
joint  design  that  was  acceptable  to  all  five 
nations,  butthe  three  signatories  did  have 
a  common  design  (9.75t)  despite  UK 
preference  for  an  lit  aircraft  (Refs:  Flight 
article,  NA  PREM  file) 
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Rating 

•  Qualitative/subjective  inputs 

•  'Polling' of  group(s) 

•  Capture  diverse  views 

•  'Cooper  Harper'  style  hierarchy 

•  Alphanumeric  output 

•  Acts  to  identify  'true  likeness' 
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The  project 
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Technologies 

•  HTML5  -  platform  agnostic/future  proof 

•  Meta  tagging  -  'Google'  type  searching 

•  Open  development  by  users 

•  Runs  secure/shared/stand-alone 

•  CD/network/online 

•  Simple/intuitive/adaptive 
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Currently  'vanilla' 
proof  of  concept 
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Users'  preferred 
flavors 
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Future 


"We  haven't  got  the  money,  so  we've  got  to 
think!"  Rutherford,  Manchester 


R.V.  Jones,  Bulletin  of  the  Institute  of  Physics  (1962)13 
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Thank  You. 


m.pryce@cranfield.ac.uk 
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